Skip to content

Add sandbox capabilities matrix benchmark and weekly workflow - #321

Open
kisernl wants to merge 11 commits into
masterfrom
devin/sandbox-capabilities-benchmark
Open

Add sandbox capabilities matrix benchmark and weekly workflow#321
kisernl wants to merge 11 commits into
masterfrom
devin/sandbox-capabilities-benchmark

Conversation

@kisernl

@kisernl kisernl commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a new sandbox-capabilities benchmark that creates one sandbox per provider and probes the full ComputeSDK surface (sandbox lifecycle, commands, filesystem, snapshots, templates, streaming, backgrounding, and port exposure). Each capability is a named ctx.step with its own try/catch so a single unsupported feature only fails that feature and does not abort the rest.

Results are aggregated per-provider into a feature matrix ({ passed, error? } per feature) and written to results/sandbox-capabilities/<YYYY-MM-DD>.json plus latest.json.

Also adds the GitHub Actions workflow:

  • Runs weekly on Monday at 10:00 UTC / 5:00 AM CDT.
  • Supports workflow_dispatch with optional provider filter, iterations, concurrency, and a publish boolean.
  • When publish is false the results are still rendered as a GitHub Actions step summary (provider × feature pass/fail table with failure details), but they are not committed back to the repo.

merge-results.ts now knows how to combine sandbox-capabilities artifacts from the per-provider matrix jobs into a single latest.json.

Link to Devin session: https://app.devin.ai/sessions/f98952f689154db886036dda3dd56bd1
Requested by: @kisernl


Open in Devin Review

Co-Authored-By: Noah Kiser <noah@computesdk.com>
@kisernl kisernl self-assigned this Aug 14, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 14, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 1 additional finding in Devin Review.

Open in Devin Review

opts: { resultsDir: string; runConfig: ResolvedRunConfig },
): Promise<void> {
const results: CapabilityMatrixResult[] = participants.map((participant) => {
const record = participant.records[0];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Capability results can report a random run instead of the first one

The capability matrix is taken from whichever run finished first (participant.records[0] at benchmarks/sandbox/capabilities-results.ts:38) rather than the first run that was started, so when more than one run per provider is requested the published matrix can come from an arbitrary run.
Impact: With more than one iteration, the saved feature matrix may reflect an arbitrary run rather than a deterministic one, making results non-reproducible.

Records arrive in completion order, so index 0 is not iteration 0

benchmarks/src/util/records.ts documents that "Platform records arrive in completion order, while the legacy result files list iterations in launch order, so every legacy bridge orders records by taskIndex" — every other writer (e.g. benchmarks/sandbox/legacy-results.ts:54) calls byTaskIndex(participant.records) first. The new writer indexes participant.records[0] directly. The config default is iterations: 1 (benchmarks/sandbox/capabilities.bench.ts:69), but the CLI can override --iterations/--concurrency, in which case records[0] is whichever task finished first and the remaining matrices are silently dropped.

Prompt for agents
In benchmarks/sandbox/capabilities-results.ts the aggregation picks participant.records[0], but platform records arrive in completion order (see the comment in benchmarks/src/util/records.ts). Other result writers normalize with byTaskIndex() before mapping. Consider ordering records by taskIndex before taking the first one (and decide explicitly how multiple iterations should be combined, e.g. first iteration or OR/AND across iterations).
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed. The writer now:

  1. Orders participant.records with byTaskIndex() before mapping them.
  2. Aggregates all per-iteration feature matrices with an AND rule: a feature only passes if every iteration passed, and the error comes from the first failing iteration.

This makes the published matrix deterministic for any --iterations override while still defaulting to the single-run case.

devin-ai-integration Bot and others added 2 commits August 14, 2026 18:22
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
@devin-ai-integration devin-ai-integration Bot changed the title Add sandbox capabilities matrix benchmark Add sandbox capabilities matrix benchmark and weekly workflow Aug 14, 2026

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 2 security concern(s).

if: env.SHOULD_RUN == 'true'
run: |
if [ "${{ github.event_name }}" = 'schedule' ]; then
pnpm update

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Scheduled benchmark runs use pnpm update instead of frozen lockfile install

Scheduled runs automatically update dependencies via pnpm update before loading vault secrets

Use pnpm install --frozen-lockfile in all CI jobs; update deps in a separate PR workflow

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/sandbox-capabilities.yml">
<violation number="1" location=".github/workflows/sandbox-capabilities.yml:92">
<priority>P1</priority>
<title>Scheduled benchmark runs use pnpm update instead of frozen lockfile install</title>
<evidence>The bench job's 'Install dependencies' step runs `pnpm update` when `github.event_name == 'schedule'`, automatically updating all dependencies to the latest semver-compatible versions without human review. Immediately after, the workflow loads all vault secrets and executes benchmark code via `npx tsx`, creating a supply-chain attack surface: a compromised dependency published between weekly runs would be automatically installed and executed with access to all provider credentials.</evidence>
<recommendation>Use `pnpm install --frozen-lockfile` unconditionally in both the `bench` and `collect` jobs. If periodic dependency updates are required, run them in a separate dedicated workflow that does not access vault secrets and does not execute benchmark code.</recommendation>
</violation>
</file>

- name: Run capabilities benchmark
if: env.SHOULD_RUN == 'true'
run: |
. benchmarks/scripts/load-vault-secrets.sh '.*'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Matrix jobs load all vault secrets via wildcard instead of provider-scoped access

Each matrix job loads all vault secrets with a wildcard regex

Pass only the current provider's name/matrix variable to the vault script

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/sandbox-capabilities.yml">
<violation number="1" location=".github/workflows/sandbox-capabilities.yml:102">
<priority>P2</priority>
<title>Matrix jobs load all vault secrets via wildcard instead of provider-scoped access</title>
<evidence>Every matrix job sources `benchmarks/scripts/load-vault-secrets.sh '.*'`, loading every secret from the vault. Since the workflow matrix spans 29 different providers, the job testing a single provider receives credentials for all other providers as well, violating least-privilege and increasing the blast radius of any provider-specific breach.</evidence>
<recommendation>Scope secret loading to only the current matrix provider by passing `${{ matrix.provider }}` or a provider-specific vault path prefix to the load script instead of the `'.*'` wildcard.</recommendation>
</violation>
</file>

devin-ai-integration Bot and others added 2 commits August 14, 2026 18:41
Co-Authored-By: Noah Kiser <noah@computesdk.com>
…s, avoid empty KEYS regex

Co-Authored-By: Noah Kiser <noah@computesdk.com>
devin-ai-integration Bot and others added 6 commits August 14, 2026 19:25
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant